Physical object boundary detection techniques and systems

ABSTRACT

Physical object boundary detection techniques and systems are described. In one example, an augmented reality module generates three dimensional point cloud data. This data describes depths at respective points within a physical environment that includes the physical object. A physical object boundary detection module is then employed to filter the point cloud data by removing points that correspond to a ground plane. The module then performs a nearest neighbor search to locate a subset of the points within the filtered point cloud data that correspond to the physical object. Based on this subset, the module projects the subset of points onto the ground plane to generate a two-dimensional boundary. The two-dimensional boundary is then extruded based on a height determined from a point having a maximum distance from the ground plane from the filtered cloud point data.

RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 17/095,589, filed on Nov. 11, 2020, which claimspriority to U.S. patent application Ser. No. 15/984,140, filed on May18, 2018, the disclosure of which is incorporated by reference herein.

BACKGROUND

Conventional techniques used to estimate shipping costs and packagingmaterials are modal and require multiple different devices, which maynot be readily available to a user. The user, for instance, may wish toship a physical object. To do so, the user first employs a tape measureto determine dimensions of the physical object. The user then determinesa size of available packaging (e.g., an envelope or box size) thatcorresponds to these dimensions, and also makes a best guess as to anamount of fill to be used in the packaging. To do so, the user mayaccess a website, from which, these materials are available, physicallyjourney to a store to purchase these materials based on the remembereddimensions, and so forth. Thus, these conventional techniques are notcapable of determining an appropriate sizing of packaging materials(e.g., box sizes), but rather are limited to finding rough dimensions ofan object, at best, using a variety of tools that may not be readilyavailable. Because of this, conventional techniques rely on a “bestguess” on the part of the user, which is inaccurate, frustrating, andcostly due to inaccuracies in obtaining excess amounts of packaging“just to be sure.” Similar frustrations may also be encountered indetermining an estimate of shipping costs of the physical object.Accordingly, these conventional techniques are inefficient and result inuser frustration.

SUMMARY

Physical object boundary detection techniques and systems are describedthat overcome the challenges of conventional techniques through improvedaccuracy achieved in calculating a three-dimensional boundary of aphysical object. This may be performed, for instance, using a devicethat is readily available to typical users, e.g., a mobile phone. Thethree-dimensional boundary may then be used to obtain shipping dataautomatically and without user intervention, e.g., shipping costs, orderpackaging, and so forth.

In one example, an augmented reality module generates three-dimensionalpoint cloud data. This data describes depths at respective points withina physical environment that includes the physical object. A physicalobject boundary detection module is then employed to filter the pointcloud data by removing points that correspond to a ground plane. A userinput is received in this example to select a location that correspondsto the physical object. The physical object boundary detection modulethen performs a nearest neighbor search to locate a subset of the pointswithin the filtered point cloud data that correspond to the physicalobject.

Based on this subset, the physical object boundary detection moduleprojects the subset of points onto the ground plane to generate atwo-dimensional boundary, e.g., a two-dimensional bounding box. Thetwo-dimensional boundary is then extruded to form a three-dimensionalboundary based on a height determined from a point having a maximumdistance from the ground plane from the filtered cloud point data.

In another example, the point cloud data is also generated by the ARmodule using a three-dimensional sensing device and point cloudgeneration module. Boundaries (e.g., rectangular boundaries) arecomputed in two dimensions for each of the perspectives, e.g., upon asurface on which the physical object is disposed. At least one of thesetwo-dimensional boundaries is selected by the module for use incomputing a three-dimensional boundary, e.g., a three-dimensionalboundary. To do so, the selected two-dimensional boundary is used togenerate a three-dimensional boundary having dimensions based on theselected two-dimensional boundary and a height of the physical objectdetected using the point cloud data.

This Summary introduces a selection of concepts in a simplified formthat are further described below in the Detailed Description. As such,this Summary is not intended to identify essential features of theclaimed subject matter, nor is it intended to be used as an aid indetermining the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. Entities represented in the figures may be indicative of one ormore entities and thus reference may be made interchangeably to singleor plural forms of the entities in the discussion.

FIG. 1 is an illustration of an environment in an example implementationthat is operable to employ techniques described herein.

FIG. 2 depicts a system in an example implementation showing operationof the AR module and physical object boundary detection module of FIG. 1in greater detail.

FIG. 3 depicts an example of point cloud data.

FIG. 4 depicts an example of filtered point cloud data generated fromthe point cloud data of FIG. 3 .

FIG. 5 depicts an example of a subset of points generated from thefiltered point cloud data of FIG. 4 using a proximity search.

FIG. 6 depicts an example of generating a two-dimensional projection ofthe subset of points of FIG. 5 .

FIG. 7 depicts an example of generating a two-dimensional boundary fromthe two-dimensional projection of FIG. 6 .

FIG. 8 depicts an example of generating a three-dimensional boundaryfrom the two-dimensional boundary of FIG. 7 using extrusion.

FIG. 9 depicts an example implementation in which the three-dimensionalboundary of FIG. 8 is displayed in a user interface as augmented realitydigital content as part of a live camera feed of the computing device ofFIG. 1 .

FIG. 10 is a flow diagram depicting a procedure in an exampleimplementation in which a three-dimensional boundary is generated basedon point cloud data obtained from an AR module.

FIG. 11 is a flow diagram depicting a procedure in an exampleimplementation in which two-dimensional boundaries are generated basedon point cloud data generated from a plurality of perspectives of aphysical object.

FIG. 12 is a flow diagram depicting a procedure in an exampleimplementation in which a two-dimensional boundary is selected from thetwo-dimensional boundaries of FIG. 11 and used to generate athree-dimensional boundary.

FIG. 13 depicts an example implementation of generation of point clouddata from a plurality of perspectives and generating two-dimensionalboundaries for each of those perspectives.

FIG. 14 illustrates an example system including various components of anexample device that can be implemented as any type of computing deviceas described and/or utilize with reference to FIGS. 1-13 to implementembodiments of the techniques described herein.

DETAILED DESCRIPTION

Overview

Physical object boundary detection techniques and systems are described.These techniques support calculation of a three-dimensional boundary fora physical object, e.g., as a three-dimensional bounding box includingdimensions of height, width, and depth. To do so, point cloud data isgenerated by an AR module of a computing device that describes depths ofpoints within a physical environment, e.g., which may be included aspart of a mobile phone and thus readily available to a user. The pointcloud data is then used by the computing device to determine athree-dimensional boundary (e.g., a bounding box) of a physical objectin the physical environment. The three-dimensional boundary may then beused to support a variety of functionality, such as to generate shippingdata to recommend packaging materials for use in packing the physicalobject (e.g., boxes, tubes, envelopes), an estimated shipping cost ofthe physical object, and may include formation of digital content thatis configured to offer the physical object for sale that includes theshipping data. In this way, the techniques described herein overcome thechallenges of conventional techniques and thus may determine appropriatepacking sizing for a physical object, rather than limited to projectionof a pre-defined box based on pre-defined measures and “best guesses” ofconventional techniques.

A user, for instance, may launch an application that is associated witha service provider system to offer products for sale. The application isconfigured to generate digital content in this example as part of theoffer for sale, e.g., a webpage, screen in a mobile application, and soon. As part of this, the application may include a user interface usableto specify a text description of the physical object, support capture ofdigital images of the physical object, and may also include shippingdata for the object.

The shipping data is generated in this example automatically and withoutuser intervention in response to a user input received via the userinterface to initiate generation of the shipping data. The shipping datamay be based on dimensions of a three-dimensional boundary that iscalculated through execution of the application by the computing device.

In a first example, an augmented reality module of the computing deviceincludes a three-dimensional sensing device and a point cloud generationmodule that are configured to generate three-dimensional point clouddata. This data describes depths at respective points within a physicalenvironment that includes the physical object, e.g., a depth map. Thesedepths may be determined using a variety of functionality, examples ofwhich include use of digital images and motion detection sensors todetermine parallax, use of radio waves (e.g., Wi-Fi waves as part ofProject Tango), time-of-flight cameras, structured light grid arrays,and so forth. In some instances, the point cloud data may have aresolution (i.e., number of points) that may cause inefficiencies inprocessing. Therefore, in such instances the point cloud data is downsampled thereby improving operation of the computing device.

A physical object boundary detection module is then employed to filterthe point cloud data by removing points that correspond to a groundplane. A user input, for instance, may be received with respect to adigital image (e.g., a live camera feed) that is displayed in a userinterface that corresponds to point cloud data being generated. The usermay then “tap” on a portion of the image that corresponds to the groundplane, e.g., a surface on which the physical object is disposed. Thephysical object boundary detection module then removes points thatcorrespond to the ground plane using a nearest neighbor search. Thisacts to isolate points above the ground plane and thus likely correspondto the physical object. In another example, the ground plane is detectedautomatically and without user intervention by the module.

Another user input is received in this example to select a location thatcorresponds to the physical object. To do so, the user then “taps” thelocation in the digital image (e.g., the live camera feed) and thephysical object boundary detection module performs a nearest neighborsearch to locate a subset of the points within the filtered point clouddata that correspond to the physical object.

Based on this subset, the physical object boundary detection moduleprojects the subset of points onto the ground plane to generate atwo-dimensional boundary, e.g., a bounding box. The two-dimensionalboundary is then extruded based on a height determined from a pointhaving a maximum distance from the ground plane from the filtered cloudpoint data to form a three-dimensional boundary. The three-dimensionalboundary thus includes dimensions that define maximum outer dimensionsof the physical object, e.g., width, depth, and height.

The three-dimensional boundary may support a variety of functionality,such as to generate shipping data that recommends packaging materials topack the physical object, estimate a shipping cost of the physicalobject, and so forth. Continuing with the previous example, the shippingdata may be included by the module automatically and without userintervention as part of the digital content offering the physical objectfor sale. In this way, the techniques described herein improveefficiency of user interaction with the computing device in a non-modalmanner in which a user may remain within a context of the application toefficiently generate the shipping costs for inclusion as part of thedigital content.

In another example, the point cloud data is also generated by the ARmodule using a three-dimensional sensing device and point cloudgeneration module. In this example, the point cloud data is generatedfrom multiple different perspectives. Two-dimensional boundaries arecomputed in two dimensions for each of the perspectives, e.g., astwo-dimensional bounding boxes upon a surface on which the physicalobject is disposed.

At least one of these two-dimensional boundaries is selected by themodule for use in computing a three-dimensional boundary. The selection,for instance, may be based on which of the two-dimensional boundaries isthe smallest, e.g., by area. The selected two-dimensional boundary isthen used to generate a bounding box or other shape (e.g., selected fromdimensions of available packaging materials) having dimensions based onthe selected two-dimensional boundary and a height of the physicalobject detected using the point cloud data to generate athree-dimensional boundary. The three-dimensional boundary may then beused to support a variety of functionality, such as to output dimensionsof the physical object, generation of augmented reality content showingthe dimensions and box, generation of shipping data, and so on. Furtherdiscussion of these and other examples is included in the followingsections and shown using corresponding figures.

In the following discussion, an example environment is first describedthat may employ the techniques described herein. Example procedures arethen described which may be performed in the example environment as wellas other environments. Consequently, performance of the exampleprocedures is not limited to the example environment and the exampleenvironment is not limited to performance of the example procedures.

Example Environment

FIG. 1 is an illustration of a digital medium environment 100 in anexample implementation that is operable to employ physical objectboundary detection techniques described herein. The illustratedenvironment 100 includes a computing device 102 illustrated as beingheld by a user 104. The computing device 102 may be configured in avariety of ways.

The computing device 102, for instance, may be configured as a desktopcomputer, a laptop computer, a mobile device (e.g., assuming a handheldconfiguration such as a tablet or mobile phone as illustrated), and soforth. Thus, the computing device 102 may range from full resourcedevices with substantial memory and processor resources (e.g., personalcomputers, game consoles) to a low-resource device with limited memoryand/or processing resources (e.g., mobile devices). Additionally,although a single computing device 102 is shown, the computing device102 may be representative of a plurality of different devices, such asmultiple servers utilized by a business to perform operations “over thecloud” as described in FIG. 14 .

In the illustrated example, the computing device 102 is configured tocapture data of a physical environment, in which, a physical object 106is disposed to generate shipping data 108. To do so, the computingdevice 102 leverages an AR module 110. The AR module 110 includes athree dimensional (3D) sensing device 112 and a point cloud generationmodule 114 that are configured to generate point cloud data 116 thatdescribes depths at respective points within the physical environment.Following a right-handed convention, for instance, the point cloud data116 may describe points in which a y-axis is vertical, and x-axis ishorizontal, and a z-axis defines depth as part of a coordinate system.

In one example, the 3D sensing device 112 and point cloud generationmodule 114 employ a technique referred to as visual-inertial odometry inwhich the 3D sensing device 112 includes motion detection sensors (e.g.,gyroscopes, accelerometers) and a camera to capture digital images ofthe physical environment. The point cloud generation module 114 isconfigured to perform computer vision analysis of the digital images torecognize features in the digital images and track differences inpositions of these features.

The point cloud generation module 114 compares motion sensing data fromthe motion detection sensors with the tracked differences in featurepositions to generate point cloud data 116 that models depths of pointsin the physical environment, e.g., as a depth map. In this way, the ARmodule 110 may track position of the computing device 102 as it moves tobuild an understanding of the physical environment based on featuresidentified from digital images. This functionality may also supportground plane detection, e.g., flat surface such as a floor or table.Thus, in this example the point cloud data 116 may be generated withoutspecialized hardware by a wide range of mobile computing devices, e.g.,smartphones and tablets.

In another example, dedicated hardware is leveraged to emit radio waves(e.g., Wi-Fi waves), the return of which is sensed (e.g., using radartechniques) by the 3D sensing device 112. This includes use of motiontracking as described above (e.g., using accelerometers, barometers,GPS, and/or gyroscopes) such that the point cloud generation module 114may gain an understanding of depths of points within the physicalenvironment in order to generate the point cloud data 116. Otherexamples are also contemplated, such as time-of-flight cameras,structured light grid array devices, and so forth.

The point cloud data 116 is then exposed by the AR module 110 via anapplication programming interface (API) to a physical object boundarydetection module 118. This module is representative of functionality togenerate boundary data 120 that describes outer dimensions of thephysical object 106 based on the point cloud data 116 as boundaries,e.g., three-dimensional polygons, tubes, and so forth.

The boundary data 120 may support a variety of functionality, such as toindicate dimensions of the physical object 106 as augmented realitydigital content as part of a live camera feed output by the computingdevice. In the illustrated example, the boundary data 120 is received asan input by a shipping module 122 to generate shipping data 108. Theshipping module 122, for instance, may communicate the boundary data 120via a network 124 to a service provider system 126. The service providersystem 126 includes a service manager module 128 that is configured togenerate the shipping data 108. The service manager module 128, forinstance, may employ a packing materials service 130 that is configuredto determine packing materials (e.g., box, tube, envelope, and fillersize) to be used to pack the physical object 106. In another instance, ashipping service 132 is configured to estimate shipping costs of thephysical object 106. In a further instance, a digital commerce service134 employs the boundary data 120 as part of generation of digitalcontent, e.g., a webpage having an offer to sell the physical object 106by the user 104.

The physical object boundary detection module 118 and shipping module122, for instance, may be incorporated as part of an application thatcorresponds to the service provider system 126. The service providersystem 126 is configured in this example to offer physical objects forsale by publishing digital content, e.g., webpages, screens in a userinterface of the application, and so on.

Accordingly, the user 104 may interact with the application to generatedigital content (e.g., a webpage, screen of an application) that offersthe physical object 106 for sale. As part of this, the shipping module122 is configured to generate shipping data 108 within a context of theapplication, and thus is non-modal. This may include automated orderingof packaging materials, listing of estimated shipping costs, and soforth which may be included as part of the digital content. In this way,generation of the boundary data 120 may improve user efficiency ininteraction with the computing device 102, and may do so in real time.

In general, functionality, features, and concepts described in relationto the examples above and below may be employed in the context of theexample procedures described in this section. Further, functionality,features, and concepts described in relation to different figures andexamples in this document may be interchanged among one another and arenot limited to implementation in the context of a particular figure orprocedure. Moreover, blocks associated with different representativeprocedures and corresponding figures herein may be applied togetherand/or combined in different ways. Thus, individual functionality,features, and concepts described in relation to different exampleenvironments, devices, components, figures, and procedures herein may beused in any suitable combinations and are not limited to the particularcombinations represented by the enumerated examples in this description.

Physical Object Boundary Detection

FIG. 2 depicts a system 200 in an example implementation showingoperation of the AR module 110 and physical object boundary detectionmodule 118 of FIG. 1 in greater detail. FIG. 10 depicts a procedure 1000in an example implementation in which a three-dimensional boundary isgenerated based on point cloud data obtained from an AR module. Althougha three-dimensional bounding box is described in this example, othershapes are also contemplated, such as tubes, polygons, triangularcolumns, envelopes, and so forth.

The following discussion describes techniques that may be implementedutilizing the described systems and devices. Aspects of the proceduremay be implemented in hardware, firmware, software, or a combinationthereof. The procedure is shown as a set of blocks that specifyoperations performed by one or more devices and are not necessarilylimited to the orders shown for performing the operations by therespective blocks. In portions of the following discussion, referencemade interchangeably to FIGS. 2 and 10 .

In this example, augmented reality functionality that is made availableon a computing device 102 (e.g., a mobile phone) is leveraged to obtainmeasurements of the physical object 106 as a three-dimensional boundary.In this instance, the three-dimensional boundary does not reconstruct ageometry of the physical object, but rather creates a minimum boundingbox for ease of packaging selection and shipping cost estimation.

To do so, point cloud data 116 is first obtained that defines depths atrespective points within a physical environment that includes thephysical object 106 (block 1002). Thus, the point cloud data 116 isconfigured as a representation of a three-dimensional physicalenvironment by the AR module 110. This may be performed in a variety ofways, such as through use of radar techniques, visual-inertial odometry,time-of-flight cameras, structured light grid techniques, and so forth.

An example 300 of point cloud data 116 is illustrated in FIG. 3 in whicha physical object 116 of a lamp is included on a table in a physicalenvironment. In some instances, the point cloud data 116 may include amultitude of points (e.g., 250,000) that may limit operation of thecomputing device. Therefore, in such instances the point cloud data 116may be down sampled (e.g., to one in twenty points per frame therebyprocessing 15,000 points) to improve operational efficiency of thecomputing device 102, e.g., to support real time output.

As previously described, the computing device 102 may leverage an ARmodule 110 to generate the point cloud data 116. The AR module 110includes a three dimensional (3D) sensing device 112 and a point cloudgeneration module 114 that are configured to generate point cloud data116 that describes depths at respective points within the physicalenvironment. In a right-handed convention, for instance, the point clouddata 116 may describe points in which a y-axis is vertical, and x-axisis horizontal, and a z-axis defines depth as part of a coordinatesystem.

In one example, the 3D sensing device 112 and point cloud generationmodule 114 employ a technique referred to as visual-inertial odometry inwhich the 3D sensing device 112 includes motion detection sensors (e.g.,gyroscopes, accelerometers) and a camera to capture digital images ofthe physical environment. The point cloud generation module 114 isconfigured to perform computer vision analysis of the digital images torecognize features in the digital images and track differences inpositions of these features.

The point cloud generation module 114 compares motion sensing data fromthe motion detection sensors with the tracked differences in featurepositions to generate point cloud data 116 that models depths of pointsin the physical environment, e.g., as a depth map. In this way, the ARmodule 110 may track position of the computing device 102 as it moves tobuild an understanding of the physical environment based on featuresidentified from digital images. This functionality may also supportground plane detection, e.g., flat surface such as a floor or table.Thus, in this example the point cloud data 116 may be generated withoutspecialized hardware by a wide range of mobile computing devices, e.g.,smartphones and tablets.

In another example, dedicated hardware is leveraged to emit radio waves(e.g., Wi-Fi waves), the return of which is sensed (e.g., using radartechniques) by the 3D sensing device 112. This includes use of motiontracking as described above (e.g., using accelerometers, barometers,GPS, and/or gyroscopes) such that the point cloud generation module 114may gain an understanding of depths of points within the physicalenvironment in order to generate the point cloud data 116. Otherexamples are also contemplated, such as time-of-flight cameras,structured light grid array devices, and so forth.

The point cloud data 116 is then filtered by a ground point filteringmodule 202 by removing points that correspond to a ground plane (block1004). A user input 204, for instance, may be received via a userinterface that displays a digital image that corresponds to the pointcloud data 116, e.g., as a live camera feed from a digital camera of thecomputing device 102. The user input 204 specifies a location of aground plane in the digital image, and thus may be correlated by theground point filtering module 202 to a ground plane in the point clouddata 116.

Points that correspond to the ground plane or lie below the ground planeare removed from the point cloud data 116 to form the filtered pointcloud data 206. This may be performed such that any points in the pointcloud data 116 are removed that have a negative dot product of theground plane within an error margin “e” as indicated by the user input204. This acts to reduce a point cloud search space to improveefficiency and operation of the computing device 102. Other instancesare also contemplated in which functionality of the ground pointfiltering module 202 is included as part of the AR module 110, itself.An example 400 of filtered point cloud data 206 is illustrated in FIG. 4.

A user input 208 is then received that specifies a location of thephysical object (block 1006) in the user interface. Continuing with theprevious example, the computing device 102 may output a live camera feedin a user interface of digital images captured of the physicalenvironment that correspond to the point cloud data 116. A user may then“tap” a location of a representation of the physical object 106 in theuser interface as illustrated in FIG. 5 .

A subset of points 212 within the filtered point cloud data 206 are thenlocated by a proximity search module 210 that correspond to the physicalobject 106 (block 1008), e.g., via a nearest neighbor search. Theproximity search module 210, for instance, may iteratively find each ofthe points in the filtered point cloud data 206 that are close to aninitial selection point “p0” of the user input 208. This is achievedusing a KD-tree to build up spatial indexing of the filtered point clouddata 206, and then query those points within a proximity “d” of theinitial object selection point “p0,” which are then added to a point set“ps.” A proximity “d” of the newly added points within the set “ps” arefurther searched in a next iteration.

As more points are discovered within proximity “d,” additional pointsare searched that are close to those newly discovered points untilpoints are no longer found and added to the set “ps,” thereby formingthe subset of points 212. As part of this, a point in the subset ofpoints 212 is found that is the furthest from the ground plane, i.e.,the highest point. An example 500 of the subset of points 212 isillustrated in FIG. 5 .

The subset of points 212 is then received as an input by atwo-dimensional projection module 214 that is configured to generate atwo-dimensional projection 216 (block 1010). The two-dimensionalprojection module 214, for instance, may reset Y coordinates to zero foreach of the points in the subset of points 212, an example 600 of whichis illustrated in FIG. 6 . This forms the two-dimensional projection 216as the subset of points 212 as disposed on the ground plane.

A two-dimensional boundary module 218 is then employed to generate atwo-dimensional boundary 220 based on the two-dimensional projection216, which may have a shape based on the projection. To do so as shownin an example 700 of FIG. 7 , a two-dimensional convex hull 702 (i.e., aboundary shape) is calculated that surrounds the points in thetwo-dimensional projection 216 as projected on the ground plane by thetwo-dimensional boundary module 218. A rotating calipers technique isthen used by the two-dimensional boundary module 218 to generate aminimum two-dimensional boundary 220 based on the two-dimensional convexhull.

The two-dimensional boundary 220 is then extruded by a three-dimensionalextrusion module 222 into a three-dimensional boundary 224 (block 1012)as the boundary data 120 in this example. As shown in the example 800 ofFIG. 8 , a two-dimensional boundary is extended into a rectangularcuboid that shares a face with the two-dimensional boundary. The heightof the three-dimensional boundary 224 is based on the point in thesubset of points 212 that is located as part of the filtering that isthe furthest from the ground plane, i.e., the highest point. Thus, thethree-dimensional boundary 224 has a shape that is based on thetwo-dimensional boundary.

Dimensions of the three-dimensional boundary are displayed in the userinterface (block 1014). As shown in the example 900 of FIG. 9 , thethree-dimensional boundary 224 is displayed as augmented reality digitalcontent within a user interface 902 including a live camera feed. Thethree-dimensional boundary 224 in this example encompasses arepresentation (e.g., displayed around) of the physical object in thedigital images.

The three-dimensional boundary 224 may be leveraged to support a varietyof functionality. Returning of FIG. 1 , for instance, boundary data 120that includes the three-dimensional boundary 224 may be communicatedover a network 124 to a service provider system 126. A service managermodule 128 of the service provider system 126 then generates shippingdata 108 based on this box. The service manager module 128, forinstance, may employ a packing materials service 140 that is configuredto determine packing materials (e.g., box, shape, envelope, and fillersize) to be used to pack the physical object 106. For example, thepacking material service 140 may select a size and shape of packagingmaterials based on dimensions of the three-dimensional boundary 224 aswell as filler (e.g., bubble wrap) to be used as filler within a volumeof the box. In this way, a correct size of the packaging materials andamounts of filler material may be determined accurately andautomatically by the computing device 102 that is readily available tothe user 104.

In another instance, a shipping service 142 is configured to estimateshipping costs of the physical object 106. The shipping service 142, forinstance, may base the estimate on dimensions of the three-dimensionalboundary. In a further instance, a digital commerce service 144 employsthe boundary data 120 as part of generation of digital content, e.g., awebpage having an offer to sell the physical object 106 by the user 104.Continuing with the previous example, a webpage may be generated tooffer the physical object 106 for sale. As part of this generation by anapplication executed by the computing device 102, the three-dimensionalboundary 224 is generated and used to generate shipping data 108, e.g.,as part of capturing digital images that are also to be included as partof the digital content from multiple perspectives. In this way, thetechniques may efficiently leverage capture of these digital images fordual purposes, thereby improving efficiency in operation of thecomputing device 102.

FIGS. 11 and 12 depict another example 1100, 1200 of physical objectboundary detection techniques and systems. In this example, a user inputis received at a computing device 102 via a user interface to initiategeneration of digital content offering a physical object 106 for salevia a network by a service provider (block 1102). The application, forinstance, may be executed by the computing device 102 as associated withthe service provider system 126. The service provider system 126includes a digital commerce service that is usable to offer products forsale. To do so, the digital content is generated to indicateavailability of the physical object 106, e.g., a webpage, user interfacescreen for output in a mobile application, and so forth.

As part of this, a user input is received at the computing device 102via the user interface of the application to initiate estimation ofshipping costs for the physical object as part of the generation of thedigital content (block 1104). Accordingly, point cloud data 116 isgenerated as previously described from an AR module 110.

As previously described, the computing device 102 may leverages an ARmodule 110 to generate the point cloud data 116. The AR module 110includes a three dimensional (3D) sensing device 112 and a point cloudgeneration module 114 that are configured to generate point cloud data116 that describes depths at respective points within the physicalenvironment. In a right-handed convention, for instance, the point clouddata 116 may describe points in which a y-axis is vertical, and x-axisis horizontal, and a z-axis defines depth as part of a coordinatesystem.

In one example, the 3D sensing device 112 and point cloud generationmodule 114 employ a technique referred to as visual-inertial odometry inwhich the 3D sensing device 112 includes motion detection sensors (e.g.,gyroscopes, accelerometers) and a camera to capture digital images ofthe physical environment. The point cloud generation module 114 isconfigured to perform computer vision analysis of the digital images torecognize features in the digital images and track differences inpositions of these features.

The point cloud generation module 114 compares motion sensing data fromthe motion detection sensors with the tracked differences in featurepositions to generate point cloud data 116 that models depths of pointsin the physical environment, e.g., as a depth map. In this way, the ARmodule 110 may track position of the computing device 102 as it moves tobuild an understanding of the physical environment based on featuresidentified from digital images. This functionality may also supportground plane detection, e.g., flat surface such as a floor or table.Thus, in this example the point cloud data 116 may be generated withoutspecialized hardware by a wide range of mobile computing devices, e.g.,smartphones and tablets.

In another example, dedicated hardware is leveraged to emit radio waves(e.g., Wi-Fi waves), the return of which is sensed (e.g., using radartechniques) by the 3D sensing device 112. This includes use of motiontracking as described above (e.g., using accelerometers, barometers,GPS, and/or gyroscopes) such that the point cloud generation module 114may gain an understanding of depths of points within the physicalenvironment in order to generate the point cloud data 116. Otherexamples are also contemplated, such as time-of-flight cameras,structured light grid array devices, and so forth.

FIG. 13 depicts an example implementation of generation of point clouddata from a plurality of perspectives and generating boundaries for eachof those perspectives. In this example, the point cloud data 116 iscaptured from a plurality of different perspectives 1302, 1304, 1306 inrelation to the physical object 106, e.g., top, side, front, back, andso forth. The point cloud data defines depths at respective pointswithin the physical environment (block 1106).

The AR module 110, for instance, may include functionality forhit-testing points to indicate a distance from a camera to a nearestobject along a vector in the camera frustum. Using these capabilities,the AR module 110 builds a one centimeter grid of points and hit-testsmultiple points each frame to build a depth map of the physicalenvironment in real time. This produces the point cloud data 116 ofpoints within the object, with points that are within a defined distancefrom a ground plane removed.

The physical object boundary detection module 118 is configured todetermine when a sufficient number of points are collected from asufficient number of perspectives, and in response, cease the generatingof the point cloud data 116 by the AR module 110. This acts to conservecomputational resources of the computing device 102. This also supportsuser feedback indicating a point in time, at which, the threedimensional bounding box may be generated.

The physical object boundary detection module 118 then computes aplurality of two-dimensional boundaries 1308, 1310, 1312, in which eachtwo-dimensional boundary corresponds to a respective one of theplurality of perspectives 1302, 1304, 1306 (block 1110). To do so, thephysical object boundary detection module 118 determines a best fit byrotating through a defined arc 1314 (e.g., ninety degrees) in increments(e.g., five degrees) and generate a two-dimensional boundary of thepoint cloud at each angle. In an implementation, the two-dimensionalboundary has a shape that corresponds to a minimum area of the points atthat perspective.

The physical object boundary detection module 118 then selects atwo-dimensional boundary from the plurality of two-dimensionalboundaries for use in estimating a bounding box (block 1202). Thephysical object boundary detection module 118, for instance, may selectthe two-dimensional boundary having the smallest (i.e., minimum) area.

The two-dimensional boundary is then used by the physical objectboundary detection module 118 to generate a three-dimensional boundaryhaving dimensions (e.g., and shape) based on the selected boundary and aheight of the physical object taken from the point cloud data (block1204). Points within the bounding box, for instance, may be comparedalong a Y axis to determine the point 1316 or cluster of points in thepoint cloud data that has the maximum distance from the ground plane1318. This defines a Y axis that, in conjunction with the size of theselected two-dimensional boundary having the minimum size define thethree-dimensional boundary.

The three-dimensional boundary is then used by the shipping module 122to obtain shipping data 108 (block 1206). Dimensions of thethree-dimensional boundary, for instance, may be compared withdimensions of packaging materials to locate a box having a minimal sizeand shape and an amount of packing (e.g., “fill”) to be used to back thephysical object 106 within the box. This may also be used to estimateshipping costs. This shipping data 108 may be displayed in a userinterface along with the three-dimensional boundary as augmented realitydigital content within the live camera feed (block 1208). The shippingdata 108 may also be included automatically and without userintervention as part of the digital content, e.g., in response to a userinput “okaying” inclusion of this data as part of the offer to sell thephysical object 106. In this way, the application supports efficientnon-modal interaction to include this data as part of the digitalcontent, and thus promotes user and operational efficiency.

Example System and Device

FIG. 14 illustrates an example system generally at 1400 that includes anexample computing device 1402 that is representative of one or morecomputing systems and/or devices that may implement the varioustechniques described herein. This is illustrated through inclusion ofthe physical object boundary detection module 118. The computing device1402 may be, for example, a server of a service provider, a deviceassociated with a client (e.g., a client device), an on-chip system,and/or any other suitable computing device or computing system.

The example computing device 1402 as illustrated includes a processingsystem 1404, one or more computer-readable media 1406, and one or moreI/O interface 1408 that are communicatively coupled, one to another.Although not shown, the computing device 1402 may further include asystem bus or other data and command transfer system that couples thevarious components, one to another. A system bus can include any one orcombination of different bus structures, such as a memory bus or memorycontroller, a peripheral bus, a universal serial bus, and/or a processoror local bus that utilizes any of a variety of bus architectures. Avariety of other examples are also contemplated, such as control anddata lines.

The processing system 1404 is representative of functionality to performone or more operations using hardware. Accordingly, the processingsystem 1404 is illustrated as including hardware element 1410 that maybe configured as processors, functional blocks, and so forth. This mayinclude implementation in hardware as an application specific integratedcircuit or other logic device formed using one or more semiconductors.The hardware elements 1410 are not limited by the materials from whichthey are formed or the processing mechanisms employed therein. Forexample, processors may be comprised of semiconductor(s) and/ortransistors (e.g., electronic integrated circuits (ICs)). In such acontext, processor-executable instructions may beelectronically-executable instructions.

The computer-readable storage media 1406 is illustrated as includingmemory/storage 1412. The memory/storage 1412 represents memory/storagecapacity associated with one or more computer-readable media. Thememory/storage component 1412 may include volatile media (such as randomaccess memory (RAM)) and/or nonvolatile media (such as read only memory(ROM), Flash memory, optical disks, magnetic disks, and so forth). Thememory/storage component 1412 may include fixed media (e.g., RAM, ROM, afixed hard drive, and so on) as well as removable media (e.g., Flashmemory, a removable hard drive, an optical disc, and so forth). Thecomputer-readable media 1406 may be configured in a variety of otherways as further described below.

Input/output interface(s) 1408 are representative of functionality toallow a user to enter commands and information to computing device 1402,and also allow information to be presented to the user and/or othercomponents or devices using various input/output devices. Examples ofinput devices include a keyboard, a cursor control device (e.g., amouse), a microphone, a scanner, touch functionality (e.g., capacitiveor other sensors that are configured to detect physical touch), a camera(e.g., which may employ visible or non-visible wavelengths such asinfrared frequencies to recognize movement as gestures that do notinvolve touch), and so forth. Examples of output devices include adisplay device (e.g., a monitor or projector), speakers, a printer, anetwork card, tactile-response device, and so forth. Thus, the computingdevice 1402 may be configured in a variety of ways as further describedbelow to support user interaction.

Various techniques may be described herein in the general context ofsoftware, hardware elements, or program modules. Generally, such modulesinclude routines, programs, objects, elements, components, datastructures, and so forth that perform particular tasks or implementparticular abstract data types. The terms “module,” “functionality,” and“component” as used herein generally represent software, firmware,hardware, or a combination thereof. The features of the techniquesdescribed herein are platform-independent, meaning that the techniquesmay be implemented on a variety of commercial computing platforms havinga variety of processors.

An implementation of the described modules and techniques may be storedon or transmitted across some form of computer-readable media. Thecomputer-readable media may include a variety of media that may beaccessed by the computing device 1402. By way of example, and notlimitation, computer-readable media may include “computer-readablestorage media” and “computer-readable signal media.”

“Computer-readable storage media” may refer to media and/or devices thatenable persistent and/or non-transitory storage of information incontrast to mere signal transmission, carrier waves, or signals per se.Thus, computer-readable storage media refers to non-signal bearingmedia. The computer-readable storage media includes hardware such asvolatile and non-volatile, removable and non-removable media and/orstorage devices implemented in a method or technology suitable forstorage of information such as computer readable instructions, datastructures, program modules, logic elements/circuits, or other data.Examples of computer-readable storage media may include, but are notlimited to, RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, digital versatile disks (DVD) or other optical storage, harddisks, magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or other storage device, tangible media, orarticle of manufacture suitable to store the desired information andwhich may be accessed by a computer.

“Computer-readable signal media” may refer to a signal-bearing mediumthat is configured to transmit instructions to the hardware of thecomputing device 1402, such as via a network. Signal media typically mayembody computer readable instructions, data structures, program modules,or other data in a modulated data signal, such as carrier waves, datasignals, or other transport mechanism. Signal media also include anyinformation delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media include wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared, and other wireless media.

As previously described, hardware elements 1410 and computer-readablemedia 1406 are representative of modules, programmable device logicand/or fixed device logic implemented in a hardware form that may beemployed in some embodiments to implement at least some aspects of thetechniques described herein, such as to perform one or moreinstructions. Hardware may include components of an integrated circuitor on-chip system, an application-specific integrated circuit (ASIC), afield-programmable gate array (FPGA), a complex programmable logicdevice (CPLD), and other implementations in silicon or other hardware.In this context, hardware may operate as a processing device thatperforms program tasks defined by instructions and/or logic embodied bythe hardware as well as a hardware utilized to store instructions forexecution, e.g., the computer-readable storage media describedpreviously.

Combinations of the foregoing may also be employed to implement varioustechniques described herein. Accordingly, software, hardware, orexecutable modules may be implemented as one or more instructions and/orlogic embodied on some form of computer-readable storage media and/or byone or more hardware elements 1410. The computing device 1402 may beconfigured to implement particular instructions and/or functionscorresponding to the software and/or hardware modules. Accordingly,implementation of a module that is executable by the computing device1402 as software may be achieved at least partially in hardware, e.g.,through use of computer-readable storage media and/or hardware elements1410 of the processing system 1404. The instructions and/or functionsmay be executable/operable by one or more articles of manufacture (forexample, one or more computing devices 1402 and/or processing systems1404) to implement techniques, modules, and examples described herein.

The techniques described herein may be supported by variousconfigurations of the computing device 1402 and are not limited to thespecific examples of the techniques described herein. This functionalitymay also be implemented all or in part through use of a distributedsystem, such as over a “cloud” 1414 via a platform 1416 as describedbelow.

The cloud 1414 includes and/or is representative of a platform 1416 forresources 1418. The platform 1416 abstracts underlying functionality ofhardware (e.g., servers) and software resources of the cloud 1414. Theresources 1418 may include applications and/or data that can be utilizedwhile computer processing is executed on servers that are remote fromthe computing device 1402. Resources 1418 can also include servicesprovided over the Internet and/or through a subscriber network, such asa cellular or Wi-Fi network.

The platform 1416 may abstract resources and functions to connect thecomputing device 1402 with other computing devices. The platform 1416may also serve to abstract scaling of resources to provide acorresponding level of scale to encountered demand for the resources1418 that are implemented via the platform 1416. Accordingly, in aninterconnected device embodiment, implementation of functionalitydescribed herein may be distributed throughout the system 1400. Forexample, the functionality may be implemented in part on the computingdevice 1402 as well as via the platform 1416 that abstracts thefunctionality of the cloud 1414.

CONCLUSION

Although the invention has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the invention defined in the appended claims is not necessarilylimited to the specific features or acts described. Rather, the specificfeatures and acts are disclosed as example forms of implementing theclaimed invention.

What is claimed is:
 1. A method implemented by at least one computingdevice, the method comprising: receiving, by the at least one computingdevice, motion sensing data and digital images corresponding to aphysical environment including a physical object; generating, by the atleast one computing device, point cloud data based on the motion sensingdata and the digital images corresponding to the physical environmentincluding the physical object; generating, by the at least one computingdevice, a three-dimensional boundary of the physical object based on thepoint cloud data generated based on the motion sensing data and thedigital images corresponding to the physical environment including thephysical object; generating, by the at least one computing device,digital content based on the three-dimensional boundary, the digitalcontent indicating an availability of the physical object; anddisplaying, by the at least one computing device, the digital contentgenerated in a user interface.
 2. The method as described in claim 1,wherein the user interface includes a display of a digital image of aphysical environment as part of a live camera feed that includes thephysical object and corresponds to the motion sensing data and thedigital images corresponding to the physical environment including thephysical object.
 3. The method as described in claim 1, wherein thedigital content is a webpage.
 4. The method as described in claim 1,wherein the digital content includes one or more of the digital images.5. The method as described in claim 1, wherein the digital content is auser interface screen for output in a mobile application.
 6. The methodas described in claim 1, wherein the point cloud data is down sampled.7. The method as described in claim 1, further comprising generating, bythe at least one computing device, shipping data for the physical objectas part of the generating of the digital content, and wherein thedigital content includes the shipping data.
 8. The method as describedin claim 7, wherein the shipping data includes an amount of packing tobe used in packaging the physical object.
 9. The method as described inclaim 7, wherein the shipping data is an estimated cost of shipping thephysical object.
 10. The method as described in claim 7, wherein theshipping data describes an estimated shipping cost based on dimensionsof the three-dimensional boundary.
 11. The method of claim 7, whereinthe digital content includes a listing of the physical object, thelisting including the shipping data generated.
 12. A system comprising:at least one hardware processor; and a non-transitory computer-readablestorage medium storing executable instructions that, when executed,cause the at least one hardware processor to perform operationscomprising: receiving motion sensing data and digital imagescorresponding to a physical environment including a physical object;generating point cloud data based on the motion sensing data and thedigital images corresponding to the physical environment including thephysical object; generating a three-dimensional boundary of the physicalobject based on the point cloud data generated based on the motionsensing data and the digital images corresponding to the physicalenvironment including the physical object; generating digital contentbased on the three-dimensional boundary, the digital content indicatingan availability of the physical object; and displaying the digitalcontent generated in a user interface.
 13. The system as described inclaim 12, wherein the digital content includes one or more of thedigital images.
 14. The system as described in claim 12, furthercomprising generating shipping data for the physical object as part ofthe generating of the digital content, and wherein the digital contentincludes the shipping data.
 15. The system as described in claim 12,wherein the digital content includes a listing of the physical object.16. A non-transitory computer-readable storage medium storing a set ofinstructions that, when executed by at least one processor, causes theat least one processor to perform operations comprising: receivingmotion sensing data and digital images corresponding to a physicalenvironment including a physical object; generating point cloud databased on the motion sensing data and the digital images corresponding tothe physical environment including the physical object; generating athree-dimensional boundary of the physical object based on the pointcloud data generated based on the motion sensing data and the digitalimages corresponding to the physical environment including the physicalobject; generating digital content based on the three-dimensionalboundary, the digital content indicating an availability of the physicalobject; and displaying the digital content generated in a userinterface.
 17. The non-transitory computer-readable storage medium asdescribed in claim 16, further comprising generating shipping data forthe physical object based on the three-dimensional boundary as part ofthe generating of the digital content, and wherein the digital contentincludes the shipping data.
 18. The non-transitory computer-readablestorage medium as described in claim 16, wherein the generating of thethree-dimensional boundary further comprises: computing a plurality oftwo-dimensional boundaries, in which each two-dimensional boundary ofthe plurality of two-dimensional boundaries corresponds to a respectiveone of a plurality of perspectives captured by the data; selecting atwo-dimensional boundary from the plurality of two-dimensionalboundaries; and generating the three-dimensional boundary havingdimensions based on the selected two-dimensional boundary and a heightof the physical object from the data.
 19. The non-transitorycomputer-readable storage medium as described in claim 16, wherein thegenerating of the three-dimensional boundary further comprises:filtering the point cloud data by removing points from the point clouddata that correspond to a ground plane; receiving, via the userinterface, a user input specifying a location of the physical object;locating a subset of points within the filtered point cloud data thatcorrespond to the physical object, the locating performed using aproximity search based on the location of the user input; generating atwo-dimensional boundary based on the located subset; and extruding thetwo-dimensional boundary into the three-dimensional boundary.
 20. Thenon-transitory computer-readable storage medium as described in claim16, wherein the generating of the three-dimensional boundary furthercomprises: computing a plurality of two-dimensional boundaries, in whicheach two-dimensional boundary corresponds to a respective one of aplurality of perspectives captured by the point cloud data; selecting atwo-dimensional boundary from the plurality of two-dimensionalboundaries; and generating the three-dimensional boundary havingdimensions based on the selected boundary and a height of the physicalobject from the point cloud data.